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(NAEP) , a survey of national trends in educational achievement, is 
attempting to expand its inclusion of students with disabilities or 
limitations that have previously caused them to be excluded from the 
assessment* The study described was a precursor to the 1996 changes 
in NAEP inclusion procedures* It was conducted in conjunction with 
the 1994 State NAEP fourth grade reading assessment. Study questions 
about exclusion and ass es sibi 1 i ty were addressed by collecting 
independent information on a four state sample of students selected 
for the 1994 fourth grade reading Trial State Assessment (TSA) who 
had also been identified by their local schools as having the 
Individualized Education Plans of special education* The final sample 
involved 416 students from 123 schools. Of these students, 185 had 
been excluded from the TSA and 231 were designated as "included," 
even though a few were absent on the day the TSA was administered. 
Results suggest that a high proportion of students with disabilities 
can and should participate in the NAEP reading assessment, but the 
current NAEP instrument does not provide sufficient accuracy in the 
lower achievement ranges to support disaggregated reports of 
proficiency levels for these students. If the goal is to achieve a 
level of measurement that would allow information about these 
students to contribute to the estimates of states' overall 
performances, the large majority of fourth graders are assessable on 
the current instrument. Suggestions are given for modifying the NAEP 
instrument. Appendixes discuss calculating the criterion for 
as s es s ib i 1 i ty and regression functions for prediction of scores. 
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1 Paper presented at the annual meeting of the American Educational Research Association, New York, 
April, 1996. The work reported herein was done under the auspices of the National Academy of Education Panel on 
the Evaluation of the NAEP Trial State Assessment and was supported by Grant No. R999A9001-90, administered 
by the Office of Educational Research and Improvement, U.S. Department of Education. 
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Introduction 



Recent educational trends, including the passage of the Goals 2000: Educate America 
Act — which calls for academic standards and assessments that are meaningful, challenging, and 
appropriate for all students, have prompted many large scale assessment programs to reevaluate 
their guidelines and procedures for assessing students with disabilities. The National Assessment 
of Educational Progress (NAEP), an important indicator of trends in national educational 
achievement since 1969, is among those seeking to expand inclusion. NAEP is a 
Congressionally-mandated survey of student achievement that assesses proficiency in core 
subject areas for national samples of students at grades 4, 8, and 12. Since 1990, NAEP, which 
is conducted by the National Center for Education Statistics (NCES) under the policy guidance of 
the National Assessment Governing Board (NAGB), has also reported state-level results for 
states participating in the voluntary State NAEP assessment program. 

Currently, NCES and NAGB, working with the NAEP contractors, have begun a number 
of changes intended to increase inclusion of students with disabilities as well as those with 
limited English proficiency (LEP students). Using a split-sample design that will assure NAEP’s 
capacity to maintain achievement trends while the new procedures are being implemented, 
assessment accommodations and bilingual Spanish-English assessment materials were made 
available to samples of students in the recently completed 1996 mathematics and science 
assessments. In addition, assessment administrators for the majority of schools in both the 
national and state NAEP programs operated under new inclusion guidelines in 1996 that are 
expected to decrease the frequency of exclusions. 

The study reported here was one precursor to these 1996 changes in NAEP inclusion 
procedures. It was conducted in conjunction with the 1994 State NAEP fourth-grade reading 
assessment, an assessment in which students were asked to read fairly lengthy texts drawn from 
actual books and stories similar to those they might encounter in class, and to demonstrate then- 
understanding of what they had read by answering a series of multiple-choice and constructed 
response questions; the format of the latter required students to write their answers in their own 
words. 2 



Carried out under the auspices of the National Academy of Education (NAE) Panel on the 
Evaluation of the NAEP Trial State Assessment (TSA) 3 , our study was intended to help NCES 



2 For a more complete description of the 1994 NAEP reading assessment, see NAEP Reading Consensus 
Project, Reading Framework for the 1992 and 1994 National Assessment of Educational Progress (Washington 
D C.. National Assessment Governing Board, 1993). Sample reading tasks from the assessment are included in J.R. 
Campbell, P.L. Donahue, C.M. Reese, and G.W. Phillips, NAEP 1994 Reading Report Card for the Nation and the 
States: Findings from the National Assessment of Educational Progress and Trial State Assessment (Washington 
D.C.: National Center for Education Statistics, January 1996). 

3 From 1990 through 1994, State NAEP assessments were authorized on a trial basis only. Beginning in 
1996, the trial status was dropped, although the State assessements remain “developmental.” 



plan for greater inclusion by providing information about the implementation of the then-current 
exclusion procedures and about the assessibility of students with disabilities in the population 
sampled by NAEP. In addition, the study informed the Congressionally-mandated NAE 
evaluation of the TSA by examining the extent to which the accuracy and fairness of State NAEP 
data might be affected by state-to- state differences in how the exclusion guidelines were applied. 

Throughout this paper we use the nominclature of the 1994 NAEP and refer to students 
with disabilities as Individualized Education Plan (IEP) students. 

Research Questions 

The 1994 IEP exclusion and assessibility study addressed four primary research 
questions: 

1 . What is the assessibility of the excluded IEP students on the NAEP reading 
assessment used in 1994? 

2. What accommodations, if any, would have been needed to include additional 
students? 

3. How was the exclusion decision process for IEP students implemented in the 
1994 TSA? 

4. Was the exclusion process implemented in a comparable manner in different 
states? More specifically, were there reading levels at which a child was likely to 
be included in some states but excluded in others? 



Methods 

The study questions were addressed by collecting independent information on a sample of 
students selected for the 1994 fourth-grade reading TSA who also had been identified by their 
local school personnel as having Individualized Education Plans. Some of these students had 
actually participated in the TSA, while others had been excluded on the basis of their disability. 
For the current study, site visitors met with the students to obtain structured measures of reading 
proficiency; they also interviewed the students’ teachers and local NAEP assessment 
administrators to gather additional information about the students and, more generally, about the 
implementation of the exclusion process. 

Sample 

As noted above, the sample for this study was based on students who had previously been 
sampled for the 1994 TSA. Sampling for the TSA began with the selection of stratified random 
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samples of approximately 100 public schools and 15 nonpublic schools per state. 4 Each 
participating school prepared a list of all students at the target grade level, and the NAEP 
contractor selected from this list a random sample of students (n » 30) and returned it to the 
school; the school, in turn, reviewed the sample to identify any withdrawn, out-of-grade, or 
otherwise ineligible students. At the same time, the school also identified any IEP or LEP 
students on the sample list and determined whether or not these identified students should be 
excluded from the TSA. 



Forty-one states participated in the 1994 TSA. 5 In order to obtain sufficient statistical 
power to address the fourth research question, concerning comparability of exclusion decisions 
across states, we concentrated the study sample into four states. Within each of these states, we 
began with a simple random sample of schools that participated in the TSA. Within sampled 
schools, we then attempted to collect information on every IEP student identified on the TSA- 
s ample list, so far as possible within the constraints of a one-day site visit. 

State sample. The four states in the sample were selected to represent high and low 
identification rates and high and low conditional exclusion rates for IEP students. That is, states 
were sorted into the IEP identification-by-exclusion matrix shown in Figure 1, using participation 
data from the 1992 TSA in fourth-grade reading. One state was then sampled from each of the 
four corner cells of the matrix. 6 



4 The sample was restricted to “regular” public and nonpublic schools and did not include special education 
schools. See J. Mazzeo, N.L. Allen, and D.L. Kline, Technical Report of the NAEP 1994 Trial State Assessment 
Program in Reading (Washington D.C.: National Center for Education Statistics, December 1995). 

Three other jurisdictions, the District of Columbia, Guam, and Department of Defense Overseas Schools, 
also participated, although the District of Columbia withdrew from the trial after data collection. 

The one exception was the high-identified/high-excluded cell, which was empty except for a state that did 
not participate in the 1994 TSA. A medium-identified/high-excluded state was sampled instead. 
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